Gallup has been polling the American public for almost 90 years and still asks many of the questions that originated in the 1930s, including presidential job approval. These and other key trended questions first asked in the 1950s, 1970s or 1990s provide an unparalleled historical record of the United States. Continuing to update the trends allows that record to endure, while enabling us to better understand the present in relation to the past.
Gallup Methodology Prioritizes Long-Term Trends
Gallup is committed to maintaining its trends for as long as possible. But the question of “how” is before us. The survey research industry has been in transition. Telephone surveys are becoming harder and more expensive to conduct, while the development of new methodologies and technologies is leading most surveys across the industry to move to self-administered modes, such as web and mail. Gallup has been a leader in the technological revolution in polling, creating the largest probability-based U.S. panel for conducting online research. We now use The Gallup Panel, as well as high-quality mail surveys, for much of our U.S. survey work.
Self-administered modes such as these have many advantages, including cost, timeliness, and convenience for respondents, all while maintaining high-quality methodology and accurate estimates. However, self-administered modes aren’t the right methodological fit for all studies, particularly for research that needs to maintain trends with long histories of interviewer-administered methodologies, such as the Gallup Poll Social Series (GPSS).
Self-administered modes and interviewer-administered modes (such as phone or face-to-face) do not always produce the same results, which is attributable to what is known as a “mode effect.” Differences in the way respondents perceive questions and response-scale choices when reading them on paper or a screen versus listening to someone else read them can greatly influence how respondents answer questions, as can the presence of an interviewer. Mode effects are well-documented in the survey research literature, and Gallup’s own research has found that these effects can be significant.
Since 2021, Gallup has conducted parallel testing of its monthly GPSS surveys by phone and web, confirming that a switch to web could produce mode effects for many of our historically trended measures. These effects would make it difficult to attribute, and clearly communicate to data users, which changes are due to true shifts in opinion versus artifacts of a switch in methodology. Accordingly, Gallup has decided to continue conducting its GPSS surveys using telephone methodology, as we continue to study these potential mode effects and how to reduce or correct for them.
Staying on the Phone, but Outsourcing
Because of the changes in surveying over the past 15 years, Gallup is investing in the development and advancement of self-administered modes for U.S. data collection. We are also reducing our investment in our in-house technology and innovation for telephone interviewing. Changes in respondent device use and call-blocking technology have led to declining contact rates, making it especially important to stay current on telephone interviewing methods and technologies to produce unbiased estimates.
Therefore, beginning in January 2024, all of Gallup’s U.S. telephone interviewing, including the GPSS, has been moved to a U.S.-based telephone data-collection partner. Gallup has chosen a partner that has large-scale telephone data-collection capabilities and has demonstrated a commitment to making the necessary investments in phone interviewing and developing mixed-mode interviewing approaches to support high-quality telephone data collection for the foreseeable future. Many of Gallup’s own telephone interviewers have made the move to this external partner with us and will continue to work on Gallup surveys.
Working with data-collection partners is not new for Gallup. We have worked with countless such partners throughout our history, both in the U.S. and worldwide. All data-collection partners execute data collection, but Gallup continues to lead all aspects of the research projects, including vetting and testing potential partners, developing the methodology and sampling frames, writing the survey instruments, training interviewers, monitoring field work, reviewing the quality of completes, weighting, and analyzing the data.
Moving surveying from one interviewing center to another can disrupt trends because of another methodological challenge called a “house effect.” This refers to the subtle, unintended differences in how the same methodology is executed by different interviewing centers, which can potentially affect the survey estimates.
Before making the switch to an interviewing partner for GPSS, Gallup conducted three months of testing -- in July, September and November 2023. During each of these months, two surveys were conducted: one in-house, using Gallup’s call center, and the other by our interviewing partner. We compared the results of the two, identified areas of slight difference and worked with our partner to adjust their procedures.
For most of the results, we observed no statistically significant differences. For others, those gaps shrank over the course of the three months as differences in call design, interviewer probing, and other aspects of the phone-room methodologies were brought into better alignment with Gallup’s approaches.
Gallup is confident in the quality of our data-collection partner. However, there is the potential that house effects could affect future results, requiring further investigation before Gallup releases data. In these cases, which are expected to be rare, Gallup may need to suppress a data point or report it without comparing it to past data on the same question.
Leaders in every important domain rely on Gallup data to be accurate and use that information to inform crucial decisions. It is therefore of the utmost importance that when we update a trend, readers can feel confident that a shift in public opinion or behavior is real and does not merely reflect changes in methodology. Gallup will continue to be transparent about methodological changes as they occur and, if warranted, any effects these changes may have on our data.
